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Theory  and  Evaluation  of  Battlefield  Visualization  in  Context 

Abstract 

This  paper  describes  an  on-going  effort  to  develop  operational  visualization  concepts 
and  their  technical  implementations  to  support  sensemaking  skills  based  on  the  Army’s 
Field  Manual  (FM)  3-0  doctrinal  information  on  “Visualization,  Detection,  and  Decide” 
requirements.  A  theory  of  visualization  is  presented  from  the  stance  that  visualization  is 
embedded  and  situated  in  human  endeavors,  and  its  cognitive  activities.  A  visualization 
tool  from  Sensemaking  Support  System  (S3)  is  used  to  evaluate  this  theory  by  using  case 
vignettes  from  a  Stability  and  Security  Operation  (SASO)  domain  to  measure 
visualization  impact  variables.  The  evaluation  metric  was  validated  with  eleven  military 
personnel  who  are  familiar  with  SASO  situations.  In  addition  to  the  inter-rater  metrics 
and  correlations  analyses,  a  linear  prediction  equation  that  relates  situation  understanding 
as  a  function  of  evidential  cues  and  Level  3  situation  awareness  is  derived.  The  strength 
of  the  model  is  demonstrated  by  its  high  correlation  factor  (R2  =83.7%)  indicating  its 
efficacy  for  use  as  a  situation  understanding  prediction  metric. 

1.0.  Introduction 

We  sometimes  do  what  we  think  or  intend,  volitionally.  We  react  to  what  we  see, 
hear,  touch,  or  feel.  Reacting  to  the  world  around  us  is  a  part  of  a  human  endeavor  that  is 
mediated  in  part  by  what  is  generally  known  as  visualization.  Many  military  related 
doctrines  have  recognized  visualization  as  a  part  of  command  and  control  (C2), 
predominantly  because  of  visualization’s  ability  to  create  agility  in  command  decision 
making.  The  Department  of  Army’s  doctrinal  handbook,  Field  Manual  (FM)  5-0  (DoD,  2005) 
summarizes  the  commander’s  role  in  exercising  C2  in  ways  that  characterizes  the  human 
endeavors.  These  are:  (1)  Visualizing  the  environment;  (2)  Describing  the  commander’s 
visualization  to  subordinates;  (3)  Directing  actions  to  achieve  results;  and  (4)  Leading  the 
command  to  accomplish  the  mission.  Collectively,  these  endeavors  are  anchored  on  many 
cognitive  processes  that  include,  but  are  not  limited  to  sense-making,  situation  awareness  and 
situation  understanding.  However,  sense-making  provides  the  basis  in  which  the  human  can  gain 
an  understanding  of  a  situation. 

Many  conceptual  discussions  of  future  force  battle  command  highlight  the 
importance  of  visualization  in  enabling  command  and  control.  For  instance,  network 
enabled  operations  are  founded  on  the  premise  that  if  the  future  force  fully  exploits  both 
shared  knowledge  (collective  visualization)  and  technical  connectivity,  then  the  resulting 
capabilities  will  dramatically  increase  mission  effectiveness  and  efficiency.  The  Army’s 
Future  Force  Capstone  Concept  for  2015-2024  (DoD,  2007a)  outlines  seven  key 
operational  ideas  that  characterize  the  capabilities  of  the  future  force;  one  of  which  is 
Network-Enabled  Battle  Command.  The  concept  asserts  that  throughout  future  campaigns 
Network-Enabled  Battle  Command  will  facilitate  the  situational  understanding  needed 
for  the  self-synchronization  and  effective  application  of  joint  and  Army  combat 
capabilities  in  any  form  of  operation. 

With  many  doctrinal  and  academic  research  literatures  reviewed,  I  argue  that 
visualization  and  cognition  are  embodied  and  situated;  externally,  it  enables  situation 
awareness,  and  internally,  it  is  controlled  by  the  mind.  It  is  the  art  and  skill  of  creating 


mental  models  of  events  or  situations — therefore,  it  requires  some  level  of  expertise 
especially  when  dealing  with  problem  solving  or  decision  making;  it  is  controlled, 
directed,  and  purposeful.  Bringing  various  concepts  on  visualization  together,  I  have 
identified  at  least  eight  ways  in  which  visualization  enables  different  human  actions  in 
situated  contexts,  especially  in  sense  making  of  complex  battlefield  information.  These 
are  identifying  system  level  constraints  and  boundaries,  entity  and  event  abstractions 
during  model  building,  identifying  decision  spaces,  state-space  representation  of 
problems  of  varying  complexities,  uncertainty  handling,  and  information  fusion  through 
sense  making,  situation  awareness,  and  situation  understanding. 

This  paper  discusses  the  applications  of  these  factors  to  the  development  of  a 
collaborative  sense  making  support  tool  with  visualization  applets  used  to  enable  human 
actions  in  battlefield  domains.  Visualization  applications  within  the  domain  of  stability 
and  security  operations  (SASO)  that  provide  real-time  dynamic  situation  awareness  are 
demonstrated.  The  implications  of  on-going  findings  to  human  factors  and  cognitive 
engineering  of  battlefield  visualization  information  are  discussed. 

2.  Theory  of  Information  Visualization 

Visualization  has  been  noted  by  psychologists  and  philosophers  (e.g.,  Searle, 
1983)  to  occur  internally  in  the  mind  and  externally  mediated  by  ecological  information 
(Gibson,  1978  ).  The  mind  itself  carries  abstract  information  such  as  thought,  ideas, 
perceptions,  feelings,  and  memories.  The  mind  is  responsible  for  shaping  of  meaningful 
spaces  for  situation  understanding  through  logical  assignments  of  concepts  and  their 
saliencies.  In  the  process  of  visualization,  the  mind  expresses  this  in  terms  of  imagination, 
precepts,  and  creativity  through  the  ability  and  capacity  to  conceive  ideas  and  concepts 
that  precede  cognition.  Figure  1  captures  this  anecdotal  explanation. 


Figure  1.  A  simple  theoretical  view  of  visualization 

As  seen  in  Figure  1  above,  visualization  is  controlled  by  both  internal  and  external 
events.  Internal  events  are  those  controlled  by  the  mind  and  other  neurological 
information  processing  elements.  The  reader  is  referred  to  Broad  (1937)  for  more 
discussions  on  mind,  consciousness,  and  visualization.  The  conscious  mind  is  responsible 
for  self  awareness.  Coupled  with  the  individual  experiences,  volitions  and  intents  (Ntuen 


and  Woodrow,  2005)  are  created.  Selective  information  representation,  recall,  and 
presentation  occur  effortlessly  in  the  mind  in  a  manner  that  internal  visualization  is 
effortless  and  automatically  controlled. 

On  the  other  hand,  external  events  are  ecologically  mediated  and  provide 
information  in  the  form  of  symbols,  signs,  and  signals  which  are  responsible  for  building 
the  mental  codes  of  the  universe  of  interest.  The  external  events  which  are  graphically 
portrayed  and  rendered  as  displays,  allow  us  to  gain  a  nominal  level  of  situation 
awareness  known  by  Endsley  (1995)  as  Level  I  situation  awareness  (SA)  because  of  the 
information  processing  at  the  sensory-perceptual  stage.  The  mapping  of  visualization  to 
the  elements  of  SA  is  constrained  by  barriers  within  the  information  boundaries.  For 
instance,  during  the  transitions  from  Level  I  SA  to  Level  II  SA,  the  comprehension  stage 
is  made  possible  by  special  filters  which  are  contributed  by  neural-cognitive  processes 
and  the  information  changes  from  outside.  The  outside  information  stimuli  mediate  most 
of  the  cognitive  activities  that  occur  internally.  Examples  include,  pattern  mapping  and 
recognition,  being  able  to  guess  or  estimate  quantity  in  a  graph,  predicting  a  system  state, 
and  so  on. 

In  the  battlefield  operational  environment,  information  is  generated,  collected,  and 
distributed  to  all  processing  agencies.  These  activities  are  externally  controlled.  However, 
the  representations  of  this  information  are  implicit  internal  activities  with  specific 
epistemic  and  representational  requirements.  Information  representation  in  any  form  or 
modality  provides  a  basis  for  human  reasoning  aimed  at  improving  the  operational 
effectiveness  of  decision  processes.  In  this  case,  the  human  reasoning  bottom-up  with 
known  data  (backward-chaining),  seeks  to  generalize  from  specific  observations. 
Retrospectively,  it  is  an  attempt  to  correlate  new  information  with  existing  ones  in  the 
long-term  memory. 

Information  from  the  operational  environment  also  provides  the  human  operators 
with  cues  responsible  for  alerting  the  operator  with  important  awareness  which  could  not 
have  been  possible  otherwise.  The  relationships  between  the  internal  and  external  events 
that  control  visualization  process  can  best  be  understood  by  understanding  the 
information  boundaries  created  by  these  neuro-cognitive  events.  The  study  in  this  area  is 
scarce. 

Building  from  the  above  explanations,  visualization  is  constructed  from  many 
cognitive  stances.  Card,  et  al.,  (1998)  define  visualization  as  a  form  of  data  representation 
that  enables  cognitive  processes  such  as  decision  making.  Koffka  (1935)  emphasizes  the 
use  of  visualization  as  a  symbolic  communication  tool — “as  a  skillful  use  of  image”.  The 
Army’s  Field  Manual  (FM)  3-0  (DoD,  2008)  views  visualization  as  a  mental  process 
towards  situation  understanding  such  as  “determining  a  desired  end  state,  and  envisioning 
how  to  move  from  one  state  of  a  system  to  another”.  Since  the  commanders  and  staffs  use 
visualization  to  look  for  knowledge  in  order  to  build  the  required  actionable  intelligence, 
our  discourse  is  focused  on  knowledge  visualization.  Knowledge  visualization  is  a  visual 
explication  of  conceptual  knowledge  through  conscious  SA  for  the  sole  purpose  of 
exploiting  the  visual  parameters,  encoding  salient  features  graphically,  providing  a  useful 
process,  and  producing  a  useful  knowledge  for  understanding  a  problem  situation. 

Card,  et  al.,  (1998)  also  observe  that  information  visualization  is  useful  to  the 
extent  that  it  increases  our  ability  to  perform  many  cognitive  activities;  and,  “the  real 
power  comes  from  devising  external  aids  that  enhance  cognitive  abilities;  for  the  sole 


purpose  of  knowledge  discovery,  decision  making,  and  explanation.”  Visualization  tools 
are  therefore  useful  in  supporting  direct  cognition  of  information  in  context  of  a  domain 
environment.  It  is  useful  for  gaining  insights  into  external  cognitions  in  context.  As  noted 
by  Eppler  and  Burkhard  (2004),  knowledge  visualization  allows  visual  representations  to 
improve  the  creation  and  transfer  of  knowledge  between  people  by  sharing  what  they 
know  and  what  they  need  to  know  through  perception  making  and  sharing.  Because  of 
the  interaction  between  the  external  and  the  human  (internal)  minds,  visualization  can 
allow  people  to  express  and  share  their  beliefs,  values,  experiences,  and  individual 
intentions.  The  sharing  of  the  commander’s  intent  with  battle  staffs  using  graphical 
displays  or  overlaid  maps  is  just  an  aspect  of  the  power  of  visualization  in  situation 
cognition.  On  this  note,  visualization  can  be  constructed  as  a  social  interaction  model  in 
which  a  group  cultural  cognition  can  emerge.  As  a  community,  the  battle  staff  can  use 
visualization  aids  to  create  and  reproduce  knowledge  through  social  relationships  and 
interactions  defined  by  common  standard  operating  pictures,  doctrines,  and  tactics, 
techniques,  and  procedures  (TTPs).  However,  notes  Novak  and  Wurst  (2004),  “in  order 
to  make  sense  out  of  information  and  construct  knowledge,  one  needs  to  contextualize  it 
within  one’s  own  existing  knowledge  and  thought  world.” 

While  visualization  tools  allow  decision  makers  to  see  cues  and  patterns  of 
information  in  space,  they  also  influence  the  use  of  two  important  types  of  cognitions. 
These  are  reflexive  and  reflective,  respectively.  Reflexive  cognition  generates  an 
automatic  response  (Shriffm  and  Schneider,  1977)  to  decision  making  because  of  an 
effortless  mapping  of  visualized  information  cues  or  patterns  to  mental  image  or  image 
carried  in  the  human  mind.  This  is  useful  in  a  non-deliberate  decision  making  where 
situations  change  rapidly  with  some  dimensions  of  chaos  and  complexity.  On  the  other 
hand,  reflective  cognition  is  the  term  used  in  cognitive  psychology  (Neisser,  1967)  to 
describe  conscious  and  thoughtful  reaction  to  stimuli.  Reflective  cognition  is  useful  for 
generating  prospective  plans  which  force  the  individual  to  adapt  a  familiar  procedure  to  a 
new  situation.  Since  the  familiar  procedure  may  not  match  the  incumbent  situation,  a  trial 
and  error  approach  is  enacted.  Reflective  cognition,  therefore,  is  the  result  of  interaction: 
our  ability  to  learn  new  things  come  from  encounters  with  the  unexpected  (that  which  lies 
outside  our  experience,  that  which  is  not  part  of  our  experiential  cognition)  that  turn  our 
path,  leading  to  new  knowledge. 

Thus,  visualization  may  not  only  constrain  the  way  we  can  interact  in  the  world, 
but  can  help  us  to  determine  the  way  the  world  appears  to  us.  In  terms  of  human  tacit 
knowledge,  visualization  may  not  necessarily  use  data — it  depends  on  how  we  form 
images  and  mental  models  of  what  we  know  through  experience.  When  computer  and 
other  technological  related  support  systems  are  used  to  render  or  display  information, 
visualization  becomes  an  aspect  of  what  Polyani  (1966)  refers  to  as  focal  knowledge. 

Both  tacit  and  focal  knowledge  are  the  major  enablers  of  sense-making  and  sense  making 
processes. 


3.  Battlefield  Visualization 


3.1  Doctrinal  Views  of  Visualization 

According  to  the  Army’s  FM  3-0  (DoD,  2008),  “Commander’s  visualization  is 
the  mental  process  of  achieving  a  clear  understanding  of  the  force’s  current  state  with 
relation  to  the  enemy  and  environment.”  Visualization  represents  an  aspect  of  embodied 
cognition  because  the  cognitive  processes  are  mitigated  with  the  environment  of 
information  displays.  These  displays  are  contextually  represented  and  rendered  to 
provide  cognitive  affordance  between  the  human  and  the  system.  As  an  embodied  artifact, 
visualization  can  serve  to  amplify  (e.g.,  elaborate  and  strengthen)  or  attenuate  (reduce) 
our  cognitive  ability.  The  general  doctrinal  structure  of  battlefield  visualization  from  the 
human  dimension  is  shown  in  Figure  2  below. 


Figure  2.  From  FM  3-0,  Figure  5-1:  Visualize,  Describe,  Direct;  pp.  5-4. 


The  commander’s  visualization  is  the  mental  process  of  achieving  a  clear 
understanding  of  the  force’s  current  state  with  relation  to  the  enemy  and  environment. 
This  is  enabled  by  good  SA,  situation  understanding,  critical  thinking  and  knowledge 
discovery  rules  and  algorithms.  In  general,  visualization  helps  in  pattern  recognition  tasks 
through  interactive  graphical  presentation  of  information.  Thus,  visualization  may  not 
only  constrain  the  way  we  can  interact  with  objects  in  the  world,  but  can  help  us  to 
determine  the  way  the  world  appears  to  us.  In  terms  of  human  tacit  knowledge, 
visualization  may  not  necessarily  use  data,  and  thus  can  be  used  represent  how  we  form 
concepts  around  our  mental  models. 


3.2  .  Battlefield  Visualization  Evaluation  in  Context 

Many  military  related  doctrines  have  recognized  visualization  as  a  part  of  C2, 
predominantly  because  of  visualization’s  ability  to  create  agility  in  command  decision 
making.  Lieutenant  General  William  S.  Wallace  (  2005)  notes  that  in  the  Battle 
Command  concept,  commanders  use  a  personal  decision-making  process  that 
incorporates  visualizing  the  operation,  describing  the  operation  in  terms  of  intent  and 
guidance,  and  then  directing  actions  within  that  intent.  FM  5-0  (DoD,  2005)  define  battle 
command  to  include  visualization  as  a  tool  to  predict  the  current  and  future  states  of  the 
battlefield;  and  the  Army  Transformation  Road  Map  (DoD,  2004)  elucidates  the  benefit 
of  visualization  in  terms  of  “seeing  and  knowing”  the  friendly  and  enemy  forces  and  then 
deciding  how  to  get  from  one  to  the  other  at  least  cost. 

Visualization  models  in  the  battlefield  should  allow  the  commanders  and  battle 
staffs  to  frame  better  hypotheses  about  the  information  in  the  environment,  and  reason 
bottom-up  or  top-down  (deduction  or  induction)  or  laterally  (abduction)  in  order  to  gain 
an  understanding  of  the  context  of  interest.  The  representative  visualization  process  often 
used  include  link  maps,  conceptual  maps,  symbols,  decision  trees,  semantic  diagrams, 
and  videos,  animations,  and  data  plots.  The  military  doctrines  emphasize  that 
commanders  should  be  able  to  visualize  a  dynamic  battlefield  with  some  accuracy  and 
use  of  the  results  to  their  decision  making  advantage.  A  commander  who  develops  this 
difficult  skill  can  reason  proactively  like  no  other.  “Seeing  the  battlefield”  allows  the 
commander  to  anticipate  and  adapt  quickly  to  changing  situations. 

The  relationship  between  sense-making  and  visualization  is  clearly  stated  in  the 
Army’s  FM  3-0  (2008)  operations  handbook  which  notes  that  visualization  is  a 
purposeful  activity  since  it  enables  people  to  detect  the  elements  of  a  situation  before 
making  any  decision.  That  is,  one  engages  in  battle  space  visualization  for  the  specific 
purpose  of  identifying  specific  actions  that  can  be  taken  to  influence  the  present  situation 
and  move  it  toward  an  intended  objective  or  end  state: 

—Commanders,  assisted  by  the  staff,  visualize  the  operation,  describe  it  in  terms  of  intent 
and  guidance,  and  direct  the  actions  of  subordinates  within  their  intent...  The  volume  of 
available  information  challenges  all  leaders.  They  assimilate  enormous  amounts  of 
information  as  they  visualize  the  operation,  describe  their  intent,  and  direct  their 
subordinates  ’  actions.  Visualizing  the  operation  is  continuous.  It  requires  commanders  to 
understand  the  current  situation,  broadly  define  the  future  situation,  assess  the  difference 
between  the  two,  and  envision  major  actions  that  link  them. 

Visualization  is  not  limited  to  any  particular  person  in  the  command  hierarchy 
since  even  the  field  soldier  must  use  visualization  skill  to  deal  with  the  “information  out 
there.”  Therefore,  battlefield  information  visualization  is  the  core  mental  process  that 
supports  decision-making  and  by  which  the  art  of  command  and  the  science  of  control  are 
realized.  For  the  battalion  commander  for  instance,  it  is  the  process  of  achieving  a  clear 
understanding  of  the  battalion's  current  state  with  relation  to  the  enemy.  These  include 
the  terrain,  mental  representation  of  the  objectives  such  as  avenues  of  approach  diagrams, 
and  determining  the  sequence  of  activities  that  moves  the  battalion  from  its  current  state 
to  the  end  state.  The  commander's  visualization  is  the  assessment  tool  throughout  the 
operation. 

At  the  C2  level,  the  commander  identifies  the  dynamics  of  opposing  forces  by 
using  a  visualization  process.  This  includes  evaluating  possible  enemy  reactions  and 


friendly  counteractions.  This  evaluation  may  lead  to  the  identification  of  possible  critical 
decision  points  throughout  the  operation.  Visualization  tools  supporting  the  commander’s 
cognitive  tasks  should  have  information  displayed  in  a  manner  that  best  supports  the 
acquisition,  exchange,  and  use  of  information.  Examples  include  the  use  of  realistic 
semiotic  representations  to  portray  terrain  information  and  information  fusion  of 
uncertain  and  disparate  information.  Three-dimensional  representation  of  information 
(i.e.,  terrain,  airspace  management,  or  weapons  engagement  envelopes)  should  be 
realistically  portrayed  (Barnes,  2003).  Information  displays  must  support  on-the-move 
operations.  Decision  oriented  graphics  symbology  should  be  displayed  clearly.  Operators 
must  have  the  ability  to  change  graphics  interactively.  Having  a  relevant  common  picture 
will  enable  the  commander  to  operate  within  the  enemy’s  decision  cycle  by 
synchronizing  forces  and  dictating  the  operational  tempo.  This  relevant  common  picture 
must  be  comprised  of  timely,  accurate,  and  relevant  friendly  and  enemy  situational  and 
status  information  laid  over  a  common,  near-real-time  representation  of  the  area  of 
operation  (including  elevation  and  natural  and  man-made  features).  Having  real-time  SA 
across  the  battlefield  will  enable  the  commander  to  intuitively  picture  the  friendly  and 
enemy  situation  and  reduce  battlefield  uncertainty  by  displaying  friendly  and  known 
enemy  force  location  and  status.  The  relevant  common  picture  must  be  scalable  to  the 
appropriate  levels  of  command,  tailored  by  functions,  and  based  on  the  user  determined 
parameters. 


4.  Performance  Evaluation  of  a  Battlefield  Visualization  Tool 

4.1.  Visualization  Performance  Factors 

Battlefield  visualization  gives  a  constant  reference  to  use  of  tacit  knowledge  that 
includes:  (a)  a  hybrid  of  covert  visualization  and  sense-making  such  as  manifested  in  the 
commander’s  intent  and  guidance;  and,  (b)  situation  awareness  which  is  guided  by  both 
external  and  semiotic  knowledge  consisting  primarily  of  ecological  sense-making  and 
focal  knowledge:  an  example  is  the  description  of  the  commander’s  priority  information 
requirement  developed  by  the  battle  staff.  Ntuen  (2009)  has  recognized  three  factors  that 
govern  these  assumptions:  (1)  Visualization  cannot  be  separated  from  the  context  in 
which  the  objects  of  displays  and  the  grounding  knowledge  for  representation  are  derived. 
(2)  Visualization  that  occurs  in  the  mind  are  equally,  if  not  more  important  to  the  physical 
phenomenon  expressed  by  and  adopted  by  situation  awareness  community.  And,  (3)  the 
organization  of  information  in  a  display  and  visualization  system  can  influence  the 
human  information  access  as  well  as  delimiting  performance.  It  is  surmised  that  any 
visualization  evaluation  metric  should  address  at  minimum,  the  information  at  each  of  the 
three  factors  above. 

Past  efforts  on  visualization  evaluation  have  focused  on  SA.  And  most  of  the 
methods  have  been  subjective  using  self-rating  techniques.  Self-rating  techniques  seek 
subjective  evidence  of  SA  by  eliciting  the  individuals’  own  self-perceptions.  Some 
examples  include,  Situational  Awareness  Behaviorally  Anchored  Rating  Scale 
(SABARS),  a  technique  used  by  expert  observers  to  rate  individuals  on  a  number  of 
observable  behaviors  related  to  SA  processes  (Strater,  et  al.,  2001).  This  approach  is  an 
extension  of  such  tools  as  the  Participant  Situation  Awareness  Questionnaire  (PSAQ) 


used  by  Matthews,  et  al.,  (2000),  the  Situational  Awareness  Rating  Tool  (SART)  of 
Taylor  (1990),  and  the  Crew  Awareness  Rating  Scale  (CARS)  described  by  McGuinness 
and  Foy  (2000).  Probe  techniques  or  query  techniques  seek  direct  evidence  of  the  content 
of  individuals’  SA.  In  this  method,  the  metrics  attempt  to  elicit  a  set  of  information  from 
the  individual  on  his  or  her  perception  and  understanding  of  the  situation,  and  then 
comparing  this  against  the  real  thing,  the  ground  truth. 

For  experimental  evaluation,  Clauser  and  Fox  (2005)  have  identified  a  framework 
(Figure  3)  that  contains  the  knowledge  facets  relevance  to  the  above  assumptions,  and 
which  also  attempts  to  capture  both  the  tacit  knowledge  (sense-making  factors)  and  the 
extrinsic  visualization  attributes.  The  information  in  the  right-hand  part  of  Figure  3  is 
added  to  illustrate  the  examples  of  cognition-visualization  embodiments.  The 
performance  variables  are  derived  from  this  representation  and  measured  in  the 
evaluation  are  shown  in  Exhibit  1 . 


Figure  3.  The  sense-making/  Information  visualization  metaphor  framework 
4.2  Subjects 

Eleven  experienced  military  officers  participated  in  the  study  voluntarily.  Four  of  the 
participants  were  picked  through  advertisement  and  personal  meetings  with  the  Army 
Reserve  Officer  Training  Corps  (ROTC)  at  North  Carolina  Agricultural  and  Technical 
State  University  (NCA&T).  Five  participants  were  from  the  civilian  (retired)  military 


population  working  at  NCA&T,  and  two  officers  from  reserved  army  component  in 
Greensboro.  All  participants  passed  the  requirements  of:  (1)  a  rank  of  Lieutenant  and 
above;  (2)  experience  commanding  a  troop  at  a  platoon  level  and  above;  and,  (3)  have 
combat  experience  in  modem  conflicts  such  as  in  Kosovo,  Gulf  war,  Iraq,  or  Afanghistan. 
All  participants  had  a  combined  military  experience  of  163  man  years  with  a  standard 
deviation  of  1 1.73  years. 


On  a  scale  of  1  to  7  (1  =  absolutely  not  useful  and  7  =  absolutely  very  useful)  give  rating  to  the  following 
items  based  on  the  situation  visualization  and  display  and  the  tasks  you  are  asked  to  perform: 

XI:  Situation  Understanding:  The  ability  to  translate  situation  information  into  actionable  knowledge  for 
decision  making. 

X2:  Evidence:  The  amount  of  evidential  cues  and  clues  provided  and  gained  during  the  visualization 
process. 

X3:  Frame  of  Reference:  The  ease  to  which  the  display  cues  support  and  enable  the  development  of 
plausible  hypotheses  related  to  the  event  causes. 

X4:  Information  Foraging:  The  ease  to  which  the  visualization  tool  helps  in  information  seeking  and 
extracting  for  sensemaking. 

X5:  Causal  Chaining:  The  ease  to  which  the  visualization  tool  helps  to  trace  the  causal  linkages  between 
the  events  and  effects. 

X6:  Team  sensemaking:  The  ease  to  which  the  visualization  tool  allows  the  team  to  collaborate. 

X7:  Level-3  SA:  The  ease  to  which  the  visualization  tool  allows  the  user  to  predict  the  future  states  of  the 
situation  and  the  effects. 

X8:  Belief  Revision:  The  extent  to  which  the  visualization  tool  helps  the  sensemaker  to  change  opinion 
and/or  revise  belief  because  of  new  information. 

Exhibit  1.  Performance  variables  used  for  visualization  tool  evaluation. 


4.3  Apparatus 

The  apparatus  consisted  of  Sensemaking  Support  System  (S3)  software  (Ntuen 
and  Gwan-Myung,  2008),  a  personal  computer,  and  1 8”  TV  monitor.  The  S3  software 
developed  for  use  as  a  team  or  an  individual  level  sensemaking.  It  is  useful  for 
sensemaking  at  the  individual  and  team  levels,  respectively.  This  collaborative 
knowledge  sharing  is  crucial  to  the  battle  staffs  that  must  collectively  connect  their 
experiences  and  share  their  perspectives  on  how  the  battlefield  information  is  related  to 
the  mission  and  the  commander’s  intent. 

The  current  features  in  S3  allows  up  to  three  users  to  conduct  sensemaking  based 
on  selected  problem  scenarios.  Each  user  logs  into  S3  with  a  protected  password  and  first 
completes  a  team  profile  survey  (TPS).  The  left  side  of  Figure  4  allows  at  least  one  user 
to  log  into  the  system.  The  right  side  shows  an  example  event  triggered  by  a  rocket 
propelled  grenade.  The  S3  display  has  different  levels  of  information  to  be  queried  by 
the  user  during  a  sensemaking  session.  The  user  can  use  a  white  board  to  draw  avenues  of 
approach  or  potential  enemy  locations.  S3  also  allows  the  user  to  search  for  information 
using  text  browsing  on  the  web.  Other  decision  information  can  be  displayed  through  use 
of  statistical  analysis  tools  on  Excel  spreadsheet. 

For  the  present  study,  a  case  on  SASO  from  Battle  Command  Battle  Laboratory 
of  Fort  Leavenworth  provides  the  vignette  that  drives  the  SASO  simulation  (SASOSIM) 
support  system.  SASOSIM  is  a  hybrid  of  S3  and  constructed  simulation  models  of 


SASO  behaviors  over  time.  SASO  has  components  for  simulating  both  the  DIME  and 
PMSEII  mapping  strategies. 1 


Figure  4.  Sample  S3  screen  captures. 


4.4  Procedure 

Each  officer  participated  in  a  team  of  two  “battle  staffs”  because  of  the  current 
limitation  of  S3  in  a  laboratory  setting.  Ideally,  there  were  55  team-  pairs.  However, 
because  of  time  schedules  and  availability  of  the  participants,  35  pair-trials  were 
completed,  but  the  post  experiment  questionnaires  were  administered  to  the  individuals 
separately,  yielding  1 1  observations  per  performance  variable  shown  in  Exhibit  1 .  The 
study  took  9  days  of  one  hour  per  session  per  team.  The  participants  was  told  to  use  the 
S3  software  to  conduct  sensemaking  exercises  based  on  a  simulated  terrorist  attack  in  a 
city  in  Iraq.  The  sensemaking  exercises  use  query  methods  to  assess  and  collect 
information  on  each  event.  For  examples,  when  an  event  occurs,  the  team  will  assess  the 
situation  to  answer  such  questions  as:  (a)  Who  is  responsible?  (b)  When  did  it  happen?  (c) 
Who  are  responsible  for  the  attack?  (d)  What  are  the  effects  (economic,  migration,  etc.)? 
(e)  What  are  the  likely  reasons  for  the  attack?  (f)  What  other  targets  are  next  possible 
candidates  for  attack?  Note  that  these  sensemaking  queries  contain  the  information 
elements  in  Figure  3.  For  instance,  question  (a)  is  related  to  assumptions  or  hypotheses 
formulation  through  framing. 

Note  that  this  paper  does  not  report  on  the  outcomes  of  the  sensemaking  process 
but  on  evaluating  the  contribution  of  the  S3  visualization  to  the  sensemaking  process.  At 
the  end  of  a  scenario,  the  participants  were  asked  to  subjectively  rate  the  effectiveness  of 
S3  as  a  visualization  tool  against  the  variables  given  in  Exhibit  1,  with  mean  rating 
statistics  in  Table  1  and  discussed  below. 

4.5  Data  Analysis 

Three  types  of  analyses  were  collected,  the  mean,  correlation,  and  regression 
coefficients.  The  first  is  the  mean  and  standard  deviations,  and  the  inter-rater  ratings 
scores  are  shown  in  Table  1  with  a  plot  in  Figure  5.  The  right  side  of  Table  1  has  the 
inter-rater  agreement  values  using  the  model  developed  by  Williamson  and  Manatunga 
(1997).  Except  for  X5,  the  causal  chaining  variable,  all  visualization  impact  variables 
show  some  agreement  with  corrected  Fisher  test  per  Williamson  and  Manatunga  (1997). 


1  [DIME  is  an  acronym  for  Diplomatic,  Informational,  Military,  and  Economics  and  PMSEII  stands  for 
Political,  Military,  Economic,  Social,  Information,  and  Infrastructure] 


That  is,  there  was  some  agreement  on  these  variables  as  measures  of  visualization  impact 
on  sensemaking. 


Table  1.  Mean  Rating  Statistics  on  Visualization  Impact  Variables 


Criterion 

Mean 

Std 

Inter-rater 

coefficient 

XI 

5.16 

1.32 

0.422 

X2 

3.83 

1.51 

0.367 

X3 

3.6 

1.33 

0.417 

X4 

5.57 

1.09 

0.503 

X5 

3.67 

1.62 

0.322a 

X6 

4.28 

1.28 

0.435 

X7 

5.93 

1.14 

0.485 

X8 

5.47 

1.05 

0.517 

anot  statistically  significant  at  p  <  0.01. 


On  the  mean  scores,  when  a  two-pair  comparison  test  with  Turkey  statistics  was 
performed  using  the  overall  mean  score  of  4.33  across  all  variables,  variables  X3  and  X5 
were  significant  at  p<  0.1  while  others  were  significantly  different  at  p  <  0.05. 


Figure  5.  Mean  scores  by  visualization  impact  variables. 

In  general,  the  following  variables  received  above  the  population  mean  score:  Level  3  SA 
X7)  was  prominent,  followed  by  information  foraging  (X4),  support  in  belief  revision 
(X8),  and  situation  understanding  (XI). The  variables  X6(  team  sensemaking)  and 
X2(seeking  evidence)  did  not  show  any  statistical  differences  in  mean  scores.  The 
Pearson  correlation  analysis  was  conducted  with  statistical  analysis  system  (SAS) 


software  as  outlined  by  Marasinghe  and  Kennedy  (2005).  The  result  of  the  correlation 
analysis  is  shown  in  Table  2. 

As  shown  in  Table  2,  there  is  no  statistical  relationship  between  how  people  frame 
a  problem  (X3)  and:  (i)  how  they  seek  information  (X4);  (ii)  the  causal  chain  reasoning 
process  used  (X5);  and,  (iii)  team  sensemaking  (X6).  Similarly,  there  is  no  evidence  of 


Table  2.  Pearson  Correlation  Coefficients  for  Visualization  Impact  Variables 


XI 

X2 

X3 

X4 

X5 

X6 

X7 

X8 

XI 

X2 

0.48 

X3 

0.61 

0.717 

X4 

0.633 

-0.416 

?? 

X5 

0.688 

0.34 

?? 

0.816 

X6 

0.739 

0.672 

-0.331 

-0.643 

?? 

X7 

0.802 

0.445 

?? 

0.381 

0.428 

0.726 

X8 

-0.575 

0.716 

-0.359 

0.353 

0.315 

-0.527 

?? 

??  indicates  non  significant  at  p  <  0.05. 

of  relationship  between  the  individual  causal  chain  reasoning  (X5)  and  team 
sensemaking;  and,  between  team  sensemaking  and  how  individuals  revise  beliefs  when 
given  new  information  (X8). 

There  were  some  striking  negative  correlations  that  need  some  explanation.  These 
are:  (a)  the  correlation  value  of  -0.416  between  evidence  and  information  foraging 
indicates  that  a  sensemaker  who  has  an  evidence  about  a  situation  may  not  be  seeking  for 
more  information;  i.e.,  as  more  evidence  is  known,  the  less  the  opportunity  to  seek  more 
information;  (b)  the  correlation  value  of -0.575  between  situation  understanding  and 
belief  revision  indicates  that  as  the  individual  achieves  a  better  situation  understanding, 
the  less  likely  that  he  or  she  will  change  an  already  hold  opinion.  This  is  a  typical  effect 
of  availability  bias  which  asserts  that  people  use  the  available  information  in  the  memory 
to  estimate  what  is  more  likely  in  a  situation(Kahneman,et  al.,  1999);  (c)  a  correlation 
value  of  -0.33 1  between  problem  framing  (X3)  and  teams  sensemaking  (X6)  indicates 
that  an  individual  has  the  tendency  to  frame  a  hypothesis  different  from  the  team  view; 
(d)  a  correlation  value  of  -0.359  between  problem  framing  (X3)  and  belief  revision 
indicates  that  individuals  may  not  likely  change  their  beliefs  once  they  are  fixed  on  a  set 
of  hypotheses.  This  is  typical  of  anchoring  bias  (Evans,  1989)  which  asserts  that  people 
have  the  tendency  to  rely  too  heavily  on  a  past  reference  retrospectively  during 
sensemaking;  it  takes  special  methods  to  disrupt  the  stereotypical  fixation;  (e)  a 
correlation  value  of  -0.643  between  information  foraging  (X4)  and  team  sensemaking 
(X6)  indicates  the  possibility  that  individual  may  not  likely  seek  new  information  once 
the  team  have  some  consensus  information  for  a  problem  situation;  and  (f)  a  correlation 
values  of  -0.527  between  team  sensemaking  (X6)  and  belief  revision  shows  the 
likelihood  that  team  members  may  not  change  their  beliefs  once  a  consensus  is  reached. 

Note  that  all  positive  correlations  indicate  some  increasing  relationship  between 
the  variables.  For  example,  a  correlation  coefficient  of  0.726  between  team  sensemaking 


(X6)  and  Level  3  SA  (X7)  confirms  the  strength  and  necessity  for  supporting  team 
sensemaking  with  common  visualization  support  (Endlsley,  2000). 

The  last  analysis  was  to  develop  a  prediction  equation  for  situation  understanding 
(XI)  as  a  function  of  evidential  cues  (X2)  produced  by  the  visualization  domain  and  the 
Level  3  SA  (X7).  Here,  situation  understanding  (SU)  is  considered  a  linear  addition  of 
SA  and  cue  (evidence  effects):  SU  =  SA  +  Cue;  in  the  current  representation,  XI  =  X7  + 
X2. 

A  linear  regression  of  the  best  fit  gives  the  model: 

XI  =  2.3  +  0.42  X2  +  0.16X7  (R2  =  0.837;  p  =  0.0003). 

The  above  relationship  is  true  for  the  rating  range  1<  X  <  7. 

5.  Summary  and  Conclusion 

This  paper  has  described  an  on-going  effort  to  develop  operational  visualization 
concepts  and  their  technical  implementations  to  support  sensemaking  skills  based  on  the 
Army’s  FM  3-0  information  on  “Visualization,  Detection,  and  Decide”  requirements. 
First,  visualization  is  our  attempt  to  allow  the  sensemakers  to  “see  the  same  thing”  in 
place  and  time  so  as  to  gain  real-time  situation  awareness.  Through  visualization,  the 
team  members  can  share  their  mental  models,  present  their  perspectives  either  textually 
or  graphically. 

A  theory  of  visualization  was  presented  from  the  stance  that  visualization  is 
embedded  and  situated  in  human  endeavors.  The  S3  visualization  tool  was  used  to 
evaluate  this  theory  by  using  case  vignettes  from  a  SASO  domain  to  measure 
visualization  impact  variables  (VIVs).  A  VIV  is  a  variable  related  to  visualization  and  the 
sensemaking  process  and  serves  as  a  moderator  to  amplify  good  sensemaking  or  impede 
the  sensemaking  process.  The  evaluation  metric  was  validated  with  eleven  military 
personnel  who  are  familiar  with  SASO  situations.  For  simplicity,  one  type  of  event  was 
evaluated.  However,  the  S3  visualization  tool  can  allow  for  experiment  with  many  types 
of  events  at  different  levels  of  complexities  (Ntuen,  2006b).  In  addition  to  the  inter-rater 
metrics  and  correlations  between  the  VIVs,  a  linear  prediction  equation  for  a  situation 
understanding  was  derived  as  a  function  of  evidential  cues  and  Level  3  SA  provided  by 
the  visualization  tool.  The  power  of  the  model  was  R2  =83.7%  indicating  its  efficacy  for 
use  as  situation  understanding  prediction  metric. 
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What  is  Visualization? 


•To  form  a  mental  image  (the  American  Heritage 
College  Dictionary). 

•The  use  of  interactive  visual  representations  of  data 
to  amplify  cognition  (Card,  etai.,  1998). 

•Skillful  use  of  images  (Koffka,  1935:  Principles  of  Gestalt 
Psychology) 

•A  mental  process  of  developing  situational 
understanding,  determining  a  desired  end  state,  and 
envisioning  how  to  move  [from  one  state  of  a  system 
to  another]-  FM3-0:Full  spectrum  operations,  DoD 
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Two  Main  Types  of  Visualization 
•Scientific  Visualization: 

□Display  of  data  using  their  statistical  (and  other 
mathematical)  properties  such  as  correlation, 
mean,  standard  deviation,  etc. 

□  Involves  both  space  and  time  orientations 


Z37! 


OD 


ID 


2D 


3D 


Isosurfaces,  volume  rendering,  and  glyphs  are  commonly  used 
techniques 

Isosurfaces  depict  the  distribution  of  certain  attributes 
Volume  rendering  allows  views  to  see  the  entire  volume  of  3-D 
data  in  a  single  image  (Nielson,  1991) 

Glyphs  provides  a  way  to  display  multiple  attributes  through 
combinations  of  various  visual  cues  (Chernoff,  1973) 
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Two  Main  Types  of  Visualization 


Scientific  Visualization: 


>Allows  analysts  to 
view  information  in 
multiple  dimensions 
and  scales. 

>Scaling  effect  may 
be  intolerant  to 
meaningfulness  of 
information  in  context 

Chlorophyll  mean  and  uncertainty 
of  the  Massachusetts  Bay  data  set 


miiihd... 


F.  Abbe:.  A.R.  Rcbinson,  R.N.  Miller,  P.J.  Haley,  vV.G. 
_H>I  b.  3.J,  Majumdfir,  A.  F=ng  and  F.  Leksea  2DD6. 
Quantifying  Uncenairiti&E  n  Ocsar  Predictions. 
Oc&a.f?QpfSDfry,  Sp&c/aJ  fssi/&  on  .-V? 

CQfFpvtatKXTiB!  Oc&sno^rsphy'  “.  PaL&zksewitz  ard  ^ 
H  =  rper.  Eds.,  Vol.  J&,  1..  S2-<C5 
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Scientific  Visualization 

-  Bertin  (1967)  identified  basic  elements  of 
diagrams  in  1967 

-  Most  early  visualization  research  focused  on 
statistical  graphs  (Card  et  al.,  1999) 

-  Data  explosion  in  1980s  (Nielson,  1991) 

-  NSF  launched  the  “Scientific  visualization” 
initiative  in  1985 

-  IEEE  1st  visualization  conference  in  1990 
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Information  Visualization: 


□  Is  the  cohesive  coupling  of  information 
characteristics  and  human  cognitive  processes 


“information  visualization”  was  first  used  in 
Robertson  et  al.  (1989) 

Early  information  visualization  systems 
emphasized 

interactivity  and  animation  (Robertson  et 
al.,  1993) 

Interfaces  to  support  dynamic  queries 
(Shneiderman,  1994) 

Layout  algorithms  (Lamping  et  al.,  1995) 


i 

► 
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Information  Visualization 


Cat-a-Con  Tree(Hearst  &  Karadi,  1997)  Visualization  Tree 

E.G.,  Social  Network 
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THEORY  OF  INFORMATION 
VISUALIZATION 
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Visualization  and  cognition  are  embodied  and  situated 


Visualization 


Situation 

Awareness 
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Information 

display 


Model 
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Visualization  and  cognition  are  embodied 
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Visualization  and  cognition  are  embodied  and  situated 

□  Embodiment 

>  A  coupling  of  perception-cognition-action  cycle 
using  sensory  information  in  the  form  of  signals, 
signs,  and  symbols. 

>  Both  visual  elements  and  cognition  form  a 
knowledge  artifact  in  context  of  task. 
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Visualization  and  cognition  are  embodied  and  situated 


□  Situated 


>  Situatedness  (Clancey,  1997;  Suchman,  1987)  holds 


that  “where  you  are,  when  you  do,  what  you  do  matters”.  Thus, 
situatedness  is  concerned  with  locating  everything  in  a  context 
so  that  the  decisions  that  are  taken  are  a  function  of  both  the 
situation  and  the  way  the  situation  in  constructed  or  interpreted. 
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THEORY  OF  INFORMATION  VISUALIZATION 
Theory  of  Mind  (ToM): 

>  Visualization  occurs  internally  in  the  mind  (Searle, 
1983) 

>  Visualization  is  externally  mediated  by  ecological 
Information  factors  (Gibson,  1978). 

>The  mind  is  responsible  for  shaping  meaningful 
spaces  for  situation  understanding. 

>The  mind  expresses  visualization  in  terms  of 
imagination,  precepts,  concepts,  ideas,  etc. 
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Internal  Visualization:  the  Theory  of  Mind  (ToM) 


I 
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Wikipedia: 

□The  Mind  collectively  refers  to  the  aspects  of  intellect 
and  consciousness  manifested  as  combinations  of 
thought,  perception,  memory,  emotion,  will  and 
imagination 

□Mind  is  often  used  to  refer  especially  to  the  thought 
processes  of  reason 

□The  mind  is  a  model  of  the  universe  built  up  from 
insights 

Thinking  involves  the  cerebral  manipulation  of 
information 
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Internal  Visualization:  the  Theory  of  Mind  (ToM) 


>lt  is  by  the  eyes  of  the  mind,  by  reasoning  over  the  whole,  by 
a  species  of  inspiration  that  the  general  sees,  knows,  and 
judges  (Napoleon  Bonaparte) 


>  Visualization  cannot  be  separated  from  the  context  in  which 
the  objects  of  displays  and  grounding  knowledge  for 
representation  are  derived  (Schneiderman). 
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External  Visualization:  Ecological  Approaches 

□“Animal  and  environment  make  an  inseparable 
pair” 

(Gibson,  1979,  p.8). 


□“What  you  see  when  you  see  a  thing  depends 
upon  what  the  thing  you  see  is”  (Fodor  &  Pylshyn, 
1981) 

Considerations  for: 


Space 

Time 

Distance 

Dynamism  such  as  movement  and  changes 
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BATTLEFIELD  VISUALIZATION 
DOCTRINAL  DRIVERS 
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The  Center  for  Human-Machine 

Studies  O 


According  to  Franks,  battle  command  means  seeing  what 
is  now,  visualizing  the  future  state  or  what  needs  to  be 
done  to  accomplish  the  mission  and  then  knowing  how  to 
get  your  organization  from  one  state  to  the  other  at  least 
cost  against  a  given  enemy  on  a  given  piece  of  terrain. 
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LTG.  William  S.  Wallace  (Military  Review,  May-June,  2005):  In  the 

Battle  Command  concept,  commanders  use  a  personal  decision-making 

process  that  incorporates  viSUdlizilig  the  OperdtiOfl,  describing 
the  operation  in  terms  of  intent  and  guidance,  and  then  directing  actions 
within  that  intent. 

Army  Transformation  Road  Map,  2003:  Battle  command  includes 

visualizing  the  current  and  desired  future  states  of  friendly  and  enemy 
forces  and  then  deciding  how  to  get  from  one  to  the  other  at  least  cost. 

FM  100-5:  Battle  command  is  the  art  of  battle  decision  making,  leading, 
motivating  soldiers  and  units  into  action.  It  includes  visualizing 

your  current  and  future  state. 
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Doctrinal  Background 


Army  FM  6-0,  Mission  Command:  Command 
and  Control  of  Army  Forces: 

Visualization  is  a  cognitive  ability  that  creates 
mental  images  based  on 

(i)  experience,  training  and  education  and 
knowledge  of  doctrines; 

(ii)  goals,  the  timetable  for  achieving  them, 
and  the  desired  end  state  to  include  mission 
and  intent;  and 

(iii)  resources  and  activities  to  achieve  the 
goals 
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How  Visualization  Enables  Human  Action  in  Situated 
Contexts:  Situation  Awareness 


Human 

endeavor 


Patterns  {search,  recognize,  etc} 


Visualization 
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Attention  {monitor,  track,  tag} 


Knowledge  discovery  {predict, 
anticipate,  relate } 


Information 
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-\ 


J 


2009  ICCRTS,  Washington,  DC,  June  15-17 


ow  Visualization  Enables  Human  Action  in  Situated 
ontexts:  Sensemaking  and  Information  Fusion 


Human 

endeavor 


Visualization 

elements 


How  things  are  connected 


Relationships 

measures 


w 


Fitting  the  puzzle 


Finding  information  to  fit  the 
context;  fitting  data  into  frame 
(Klein,  1998). 
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Information  fusion  for  common 
metric 


Information  integration  from 
multi-dimensional  scales 


Creating  knowledge 


Using  data  to  obtain  information; 
information  processed  into 
knowledge 
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EXPERIMENTAL  STUDY 


Visualization  Performance  Factors  (VPF) 

1  .Reference  to  a  hybrid  of  covert  visualization  (ToM)  and 
tacit  knowledge  (sensemaking) 

2.  Situation  awareness  guided  by  external  and  semiotic 
knowledge  (information  displays,  symbols,  signs,  signals) 

Objective: 

□Identity  VPF  and  the  relationships. 


Approach: 

□Subjective  data  collection.  Anecdotal  and  proof-of- 
concept 


41 

£ 
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EXPERIMENTAL  STUDY 

Past  Studies 

]  1.  Focus  on  situation  awareness 
2. Most  study  utilize  self-rating  subjective  scales 

1. E.g.,  SABARS  (Situation  Awareness 
behaviorally  Anchored  Rating  Scales — 
Strater,  et  al.,  2001) 

2.  PSAQ  (Participant  Situation  Awareness 
Questionnaire — Mathews,  et.,  2000) 

3.SART  (Situation  Awareness  Rating  Tool 
(Taylor,  1990) 
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APPARATUS 

SASOSIM:  Stability  and  Security 
Operation  Simulation 

1  .A  simulation  model  developed  from 
operational  vignettes  from  Fort 
Leavenworth. 

2.  Run  on  Sensemaking  Support  System 
(S3)  environment. 

3. AIIOWS  a  single  or  multiple  users  (up  to  5) 
at  the  same  time. 
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APPARATUS 

Sensemaking  Support  System 
(S3)  Visualization  Software  Tool 
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Najaf.? 
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S3  Allows  for  Terrain  Visualization  Using 
Google  Earth  Map 


Centric  C2  |  Sensemaking  Support  System  (S3) 
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S3  Creates  Retrospective  Information  Linkages 
(Right),  and  Allows  the  User  to  Use  a  Whiteboard  to 
Mark  Areas  of  Interest  (Left) 
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Participants: 

1 1  volunteered  military  officers 

4  Army  Reserve  Training  Corps  (ROTC)  from 
North  Carolina  A&T  State  University 

5  Civilian  (retired  military)  working  at  the 
university  +  Army 

2.  Reserve  component  in  Greensboro 

Combined  military  experience  =  163  man  years  (std= 
11.73) 

Requirements: 

>A  rank  of  Lieutenant  &  above 

>  Experience  as  a  commander  from  a  platoon  level  and 

above 

>Have  combat  experience  in  modern  conflicts  such  as 
Iraq. 
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Approach  to  VPF  Using  Clauser  and  Fox  Method 
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(1)  A  prior  information  in  the  form  of  texts,  transcripts, 
videos,  voice,  etc:  e.g.,  Al-Qaida  footprints  from 
satellite  photos 

(2)  A  set  of  hypotheses  indicating  other  possible  causal 
cues 

(3)  The  types  of  weapons  used  and  the  locations  of 
attacks 

(4)  Preaching  in  the  mosque,  staying  home  on  a  market 
day  by  some  groups;  Recruiting  around  the  areas  in 
which  attacks  occur. 

(5)  Mapping  similar  attack  behaviors  and  profiles  in 
different  austere  regions. 

(6)  Determining  some  clues  about  the  states  of 
agitation  and  pandemonium;  Estimating  the 
likelihood  of  volatile  areas  being  attacked  while 
ignoring  possible  attacks  on  stable  regions. 

(7)  Uncertainties  associated  with  temporal  events  and 
processes.  E.g.  unpredictable  hit  and  run  by  sniper 
weapons,  EIDs,  and  kidnapping. 
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Procedure: 

>Create  a  team  of  2  subjects  representing  battlestaffs. 

>Possible  55-team  pairs  (1 1  permuted  by  2)!! 

>35  pair-trials  used  due  to  scheduling  problem 
>Post  experiment  questionnaires  administered  to 
individuals  separately. 

>The  study  took  9  days  of  1  hour  per  team 
>The  participants  receive  training  on  SASOSIM  for 
sensemaking  process. 

> Events  requiring  emergency  response  were  created 
(e.g,  bombing,  EID  attack,  etc)  -see  next  slide. 

>The  team  assessed  the  situation  on  each  event: 
>Who  is  responsible? 

>When  did  it  happen? 

>Who  are  responsible? 

>What  are  anticipated  effects? 

>What  are  other  likely  targets 
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The  Center  for  Human-Machine 

Studies 
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The  Center  for  Human-Machine 


Visualization  Performance  Factors  Analyzed — Post 
Experiment  Survey 

On  a  scale  of  1  to  7  (1  =  absolutely  not  useful  and  7  =  absolutely  very  useful)  give 
rating  to  the  following  items  based  on  the  situation  visualization  and  display  and  the 
tasks  you  are  asked  to  perform: 

XI:  Situation  Understanding:  The  ability  to  translate  situation  information  into 
actionable  knowledge  for  decision  making. 

X2:  Evidence:  The  amount  of  evidential  cues  and  clues  provided  and  gained  during 
the  visualization  process. 

X3:  Frame  of  Reference:  The  ease  to  which  the  display  cues  support  and  enable  the 
development  of  plausible  hypotheses  related  to  the  event  causes. 

X4:  Information  Foraging:  The  ease  to  which  the  visualization  tool  helps  in 
information  seeking  and  extracting  for  sensemaking. 

X5:  Causal  Chaining:  The  ease  to  which  the  visualization  tool  helps  to  trace  the 
causal  linkages  between  the  events  and  effects. 

X6:  Team  sensemaking:  The  ease  to  which  the  visualization  tool  allows  the  team  to 
collaborate. 

X7:  Level-3  SA:  The  ease  to  which  the  visualization  tool  allows  the  user  to  predict 
the  future  states  of  the  situation  and  the  effects. 

X8:  Belief  Revision:  The  extent  to  which  the  visualization  tool  helps  the  sensemaker 
to  change  opinion  and/or  revise  belief  because  of  new  information. 
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RESULTS 


Three  types  of  analyses: 

I.Mean,  standard 
deviations, 

and  inter-rater  agreement 
(Williamson  &  Manatunga, 
1997) 

Except  for  causal  chaining 
variable,  all  VPF  show  some 
agreement  with  corrected 
Fisher  test  criterion —  the 
subjects  did  not  agree  on 
the  variable  as  a  metric  for 
VPF. 
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Criterion 

Mean 

Std 

Inter-rater 

coefficient 

SU  (XI) 

5.16 

1.32 

0.422 

Evidence 

(X2) 

3.83 

1.51 

0.367 

FoF (X3) 

3.6 

1.33 

0.417 

Info. 

Forage 

(X4) 

5.57 

1.09 

0.503 

CC  (X5) 

3.67 

1.62 

0.322a 

Team 

(X6) 

4.28 

1.28 

0.435 

SA- 

3(X7) 

5.93 

1.14 

0.485 

Belief 

(X8) 

5.47 

1.05 

0.517 

a:  not  statistically  significant  at  p  <0.01 


RESULTS 


Three  types  of  analyses: 

1  .With  a  two-pair  Turkey  test 
using  the  overall  mean  of 
4.33  across  all  variables: 
Frame  of  reference  and 
causal  chaining  were  on 
significant  at  p  <  0.01; 

All  other  PVF  were 
significant  at  p  <  0.05. 

Level  III  SA  was  prominently 
different  indicating  strong 
visualization  measure;  and 
so  were  information  foraging 
and  contributions  to  belief 
revision 
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■  Series! 


Visualization  Impact  Variables 


RESULTS 


XI 

X2 

X3 

X4 

X5 

X6 

X7 

X8 

XI 

X2 

0.48 

X3 

0.61 

0.717 

X4 

0.633 

-0.416 

?? 

X5 

0.688 

0.34 

?? 

0.816 

X6 

0.739 

0.672 

-0.331 

-0.643 

?? 

X7 

0.802 

0.445 

?? 

0.381 

0.428 

0.726 

X8 

-0.575 

0.716 

-0.359 

0.353 

0.315 

-0.527 

?? 

??  Indicates  non  significant  at  p  <  0.05 
2.  Correlation  Analysis: 


No  statistical  relationship  between  how  people  frame  a  problem 
and:  (1)  how  they  seek  information;  (2)  the  causal  chain  process 
used;  and  (3)  team  sensemaking. 

Negative  correlations:  -0.416  between  evidence  and  information 
raging  indicates  that  there  is  no  need  for  seeking  further 
information  once  evidence  is  known. 

Positive  correlations:  Indicates  increasing  relationship  between 
variables 

2009  ICCRTS,  Washington,  DC,  June  15-17 


RESULTS 

3.  Prediction  Equation  for  Situation 
Understanding: 


SU  (XI)  =  2.3  +  0.42  Clues  from  SA  (X2)  + 
0.16  Level  III  SA  (X7) 

(1  <  {XI ,  X2,  X7}<7) 
p  =  0.0003 
R2  =  0.837 


01 

£ 
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SUMMARY  AND  CONCLUSION 

Evaluation  study  is  preliminary.  There  is  an  on-going  study  to 
develop  a  metric  for  sensemaking  and  visualization 

Some  notables: 

The  correlation  value  of -0.575  between  situation  understanding 
and  belief  revision  indicates  that  as  the  individual  achieves  a 
better  SU,  the  less  likely  that  he/she  will  change  an  already  hold 
opinion — pointing  to  availability  bias  which  asserts  that  people 
use  the  available  information  in  the  memory  to  estimate  what  is 
more  likely  in  a  situation  (Kahneman,  et.  al.,  1999). 

Individuals  may  NOT  likely  to  change  their  beliefs  once  they  are 
fixed  on  a  set  of  hypotheses — confirming  anchoring  bias  (Evans, 
1989)  which  assert  that  people  have  the  tendency  to  rely  too 
heavily  on  retrospective  knowledge  during  sensemaking. 

Teams  will  NOT  seek  for  further  information  once  a  consensus 
has  been  reached  (-0.643  between  information  foraging  and 
team  sensemaking). 
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